{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "Choosing a performance estimation method often depends on the data one is modelling. For example, when one can assume independence and an identical distribution(i.i.d.) among observations, **cross-validation** is typically the most appropriate method[1].\n",
    "\n",
    "In the general machine learning modeling process, we usually use the k-fold cross-validation method of Fig.(a). However, time series data has temporal autocorrelation, so the test data cannot appear before the train data, otherwise it will cause data leakage！\n",
    "\n",
    "Therefore, we have 3 built-in cross-validation strategies for time series forecasting tasks in HyperTS.\n",
    "\n",
    "- First, as fig.(b) *preq-bls*: The data is split into n blocks. In the initial iteration, only the first two blocks are used, the first for training and the second for test. In the next iteration, the second block is merged with the first and the third block is used for test. This procedure continues until all blocks are tested.\n",
    "\n",
    "- Second, as fig.(c) *preq-slid-bls*: Instead of merging the blocks after each iteration (growing window), one can forget the older blocks in a sliding window fashion. This idea is typically adopted when past data becomes deprecated, which is common in non-stationary environments.\n",
    "\n",
    "- Third, as Fig.(d) *preq-bls-gap*: This illustrates a prequential approach applied in blocks, where a gap block is introduced. The rationale behind this idea is to increase the independence between training and test sets.\n",
    "\n",
    "For time series classification and regression, because they are splited at the sample level, information leakage is not involved. We adopt the cross-validation method in Fig.(a)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![cv](./images/cv.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References\n",
    "\n",
    "[1] Cerqueira V, Torgo L, Mozetič I. Evaluating time series forecasting models: An empirical study on performance estimation methods[J]. Machine Learning, 2020, 109(11): 1997-2028."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example of use:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperts.datasets import load_network_traffic\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = load_network_traffic(univariate=True)\n",
    "train_data, test_data = train_test_split(df, test_size=168, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Create Experiment and Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperts import make_experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the parameter ``cv=True``, ``num_folds`` controls the number of folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = make_experiment(train_data=train_data.copy(),\n",
    "                             task='forecast',\n",
    "                             mode='dl',\n",
    "                             timestamp='TimeStamp',\n",
    "                             covariates=['HourSin', 'WeekCos', 'CBWD'],\n",
    "                             forecast_train_data_periods=24*12,\n",
    "                             max_trials=5,\n",
    "                             cv=True,\n",
    "                             num_folds=3)\n",
    "model = experiment.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method Pipeline.get_params of Pipeline(steps=[('data_preprocessing',\n",
       "                 TSFDataPreprocessStep(covariate_cleaner=CovariateTransformer(covariables=['HourSin',\n",
       "                                                                                           'WeekCos',\n",
       "                                                                                           'CBWD'],\n",
       "                                                                              data_cleaner_args={'correct_object_dtype': False,\n",
       "                                                                                                 'int_convert_to': 'str'}),\n",
       "                                       covariate_cleaner__covariables=['HourSin',\n",
       "                                                                       'WeekCos',\n",
       "                                                                       'CBWD'],\n",
       "                                       covariate_cleaner__data_cleaner_args={'correct_object_dtype': False,\n",
       "                                                                             'int_convert_to': 'str'},\n",
       "                                       covariate_cols=['HourSin', 'WeekCos',\n",
       "                                                       'CBWD'],\n",
       "                                       cv=True, freq='H',\n",
       "                                       name='data_preprocessing',\n",
       "                                       timestamp_col=['TimeStamp'],\n",
       "                                       train_data_periods=288)),\n",
       "                ('estimator',\n",
       "                 <hyperts.hyper_ts.HyperTSEstimator object at 0x00000287E696A550>)])>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Infer and Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Metirc</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mae</td>\n",
       "      <td>0.5523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mse</td>\n",
       "      <td>0.4975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>rmse</td>\n",
       "      <td>0.7053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mape</td>\n",
       "      <td>0.3192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>smape</td>\n",
       "      <td>0.2533</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Metirc  Score\n",
       "0    mae 0.5523\n",
       "1    mse 0.4975\n",
       "2   rmse 0.7053\n",
       "3   mape 0.3192\n",
       "4  smape 0.2533"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test, y_test = model.split_X_y(test_data.copy())\n",
    "forecast = model.predict(X_test)\n",
    "results = model.evaluate(y_true=y_test, y_pred=forecast)\n",
    "results"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
